Synthesizing Generic Experimental Environments for Simulation

International audienceExperiments play an important role in parallel and distributed computing. Simulation is a common experimental technique that relies on abstractions of the tested application and execution environment but offers reproducibility of results and fast exploration of numerous scenarios. This article focuses on setting up the experimental environment of a simulation run. First we analyze the requirements expressed by different research communities. As the existing tools of the literature are too specific, we then propose a more generic experimental environment synthesizer called SIMULACRUM. This tool allows its users to select a model of a currently deployed computing grid or generate a random environment. Then the user can extract a subset of it that fulfills his/her requirements. Finally the user can export the corresponding XML representation

Towards Scalable, Accurate, and Usable Simulations of Distributed Applications and Systems

The study of parallel and distributed applications and platforms, whether in the cluster, grid, peer-to-peer, volunteer, or cloud computing domain, often mandates empirical evaluation of proposed algorithm and system solutions via simulation. Unlike direct experimentation via an application deployment on a real-world testbed, simulation enables fully repeatable and configurable experiments that can often be conducted quickly for arbitrary hypothetical scenarios. In spite of these promises, current simulation practice is often not conducive to obtaining scientifically sound results. State-of-the-art simulators are often not validated and their accuracy is unknown. Furthermore, due to the lack of accepted simulation frameworks and of transparent simulation methodologies, published simulation results are rarely reproducible. We highlight recent advances made in the context of the SimGrid simulation framework in a view to addressing this predicament across the aforementioned domains. These advances, which pertain both to science and engineering, together lead to unprecedented combinations of simulation accuracy and scalability, allowing the user to trade off one for the other. They also enhance simulation usability and reusability so as to promote an Open Science approach for simulation-based research in the field.L'étude de systèmes et applications parallèles et distribués, qu'il s'agisse de clusters, de grilles, de systèmes pair-à-pair de volunteer computing, ou de cloud, demandent souvent l'évaluation empirique par simulation des algorithmes et solutions proposés. Contrairement à l'expérimentation directe par déploiement d'applications sur des plates-formes réelles, la simulation permet des expériences reproductibles pouvant être menée rapidement sur n'importe quel scénario hypothétique. Malgré ces avantages théoriques, les pratiques actuelles en matière de simulation ne permettent souvent pas d'obtenir des résultats scientifiquement éprouvés. Les simulateurs classiques sont trop souvent validés et leur réalisme n'est pas démontré. De plus, le manque d'environnements de simulation communément acceptés et de méthodologies classiques de simulation font que les résultats publiés grâce à cette approche sont rarement reproductibles par la communauté. Nous présentons dans cet article les avancées récentes dans le contexte de l'environnement SimGrid pour répondre à ces difficultés. Ces avancées, comprenant à la fois des aspects techniques et scientifiques, rendent possible une combinaison inégalée de réalisme et précision de simulation et d'extensibilité. Cela permet aux utilisateurs de choisir le grain des modèles utilisés pour ses simulations en fonction de ses besoins de réalisme et d'extensibilité. Les travaux présentés ici améliorent également l'utilisabilité et la réutilisabilité de façon à promouvoir l'approche d'Open Science pour les recherches basées sur la simulation dans notre domaine

MINTCar : A tool for multiple source multiple destination network tomography

International audienceIdentifying and inferring performances of a network topology is a well known problem. Achieving this by using only end-to-end measurements at the application level is known as network tomography. When the topology produced reflects capacities of sets of links with respect to a metric, the topology is called a Metric-Induced Network Topology (MINT). Tomography producing MINT has been widely used in order to predict performances of communications between clients and server. Nowadays grids connect up to thousands communicating resources that may interact in a partially or totally coordinated way. Consequently, applications running upon this kind of platform often involve massively concurrent bulk data transfers. This implies that the client/server model is no longer valid. In this paper, we present MINTCar, a tool which is able to discover metric induced network topology using only end-to-end measurements for paths that do not necessarily share neither a common source nor a common destination

Toward a Formal Multiscale Architectural Framework for Emerging Properties Analysis in Systems of Systems

International audienceSystems of systems (SoSs) are composed of multiple operationally and managerially independent systems whose cooperation may lead to the apparition of emergent behaviors. Analysis of constituents and emergent properties is a cornerstone of SoSs. However, systems constituting a SoS may be arbitrarily complex. Therefore, precise modeling of SoSs may produce an enormous amount of information. Analyzing and modeling SoSs is thus a difficult task: how to conciliate complexity and provability of existence/absence of (emergent) properties ? Multiscale architecture modeling is appropriate for handling the inherent complexity of SoSs. Multiscale modeling enables to look at a problem simultaneously from different scales and different levels of detail. It takes advantage of data available at distinct scales, accordingly managing the complexity of behavior involved. Existing works regarding multiscale software architecture often specify a set of fixed views with loose definition of scales and scale dimensions, dramatically restricting scales usage. Furthermore, specification of scale changes have been mildly studied and are often handled as simple refinements. Yet, an adequate representation of model transformations is a key-factor for enabling system analysis. In this paper, we firstly present the formal definition of two scale dimensions: extend and grain. Extend allows to flexibly consider various subsystems, while grain specify different level of details. We formally define scale changes in this context and study their impact on system (emergent) propertie

Tomographie depuis plusieurs sources vers de multiples destinations dans les réseaux de grilles informatiques hautes performances

AIX-MARSEILLE2-BU Sci.Luminy (130552106) / SudocSudocFranceF

An Autonomic Cloud Management System for Enforcing Security and Assurance Properties

International audienceEnforcing security properties in a Cloud is a difficult task, which requires expertise. However, it is not the only security-related challenge met by a company migrating to a Cloud environment. Indeed, the tenant must also have assurance that the requested security properties have effectively been enforced. Therefore, the Cloud provider has to offer a way of monitoring the security. In this paper, we present a solution to express the assurance properties based on the security requirements of the tenant and to deploy these assurance properties. First, we introduce a language that expresses the assurance based on the tenant's security requirements. Secondly, we propose an infrastructure that deploys the assurance in a Cloud environment. This solution aims to be easy to use: the assurance directly results from the high-level expression of the tenant's security requirements, and no additional action is needed from the tenant. Consequently, we address one of the greatest drawback of security and assurance - the complexity of their configuration - while providing a complete assurance mechanism

Shortest Processing Time First and Hadoop

International audienceBig data has revealed itself as a powerful tool for many sectors ranging from science to business. Distributed data-parallel computing is then common nowadays: using a large number of computing and storage resources makes possible data processing of a yet unknown scale. But to develop large-scale distributed big data processing, one have to tackle many challenges. One of the most complex is scheduling. As it is known to be an optimal online scheduling policy when it comes to minimize the average flowtime, Shortest Processing Time First (SPT) is a classic scheduling policy used in many systems. We then decided to integrate this policy into Hadoop, a framework for big data processing, and realize an implementation prototype. This paper describes this integration, as well as tests results obtained on our testbed

Scalable Multi-Purpose Network Representation for Large Scale Distributed System Simulation

International audienceConducting experiments in large-scale distributed systems is usually time-consuming and labor-intensive. Uncontrolled external load variation prevents to reproduce experiments and such systems are often not available to the purpose of research experiments, e.g., production or yet to deploy systems. Hence, many researchers in the area of distributed computing rely on simulation to perform their studies. However, the simulation of large-scale computing systems raises several scalability issues, in terms of speed and memory. Indeed, such systems now comprise millions of hosts interconnected through a complex network and run billions of processes. Hence, most simulators trade accuracy for speed and rely on very simple and easy to implement models. However, the assumptions underlying these models are often questionable, especially when it comes to network modeling. In this paper, we show that, despite a widespread belief in the community, achieving high scalability does not necessarily require to resort to overly simple models and ignore important phenomena. We show that relying on a modular and hierarchical platform representation, while taking advantage of regularity when possible, allows us to model systems such as data and computing centers, peer-to-peer networks, grids, or clouds in a scalable way. This approach has been integrated into the open-source SimGrid simulation toolkit. We show that our solution allows us to model such systems much more accurately than other state-of-the-art simulators without trading for simulation speed. SimGrid is even sometimes orders of magnitude faster.La réalisation d'expériences pour l'étude de systèmes de calcul distribués à grande échelle est généralement délicate et très consommatrice de temps. Les variations non contrôlées de la charge externe empêchent de reproduire les expériences et de tels systèmes (par exemple dans le cas de plates-formes de production ou bien de systèmes en cours de conception) ne sont généralement pas disponibles pour la conduite d'expériences à des fins de recherche en informatique. C'est pourquoi de nombreux chercheurs dans le domaine du calcul distribué basent leurs études sur des simulations. Cependant, la simulation d'un système de calcul distribué à grande échelle soulève à son tour de nombreuses difficultés, notamment en terme de vitesse et d'espace mémoire. En effet, de tels systèmes sont couramment constitués de millions d'hôtes interconnectés par un réseau complexe et sur lesquels s'exécutent des milliards de processus. La plupart des simulations troquent de la précision pour de la vitesse et se reposent sur des modèles simplistes et qui peuvent être mis en oeuvre très efficacement. Néanmoins les hypothèses sous-jacentes à ces modèles sont souvent très discutables, en particulier en ce qui concerne la modélisation du réseau. Dans ce rapport, nous coupons court à l'idée largement répandue dans notre communauté selon laquelle le passage à l'échelle des simulation se ferait nécessairement en ayant recours à des modèles extrêmement simplistes et en ignorant des phénomènes potentiellement importants. Nous montrons qu'en utilisant une représentation modulaire et hiérarchique de la plate-forme tout en tirant parti de ses régularités quand elles sont présentes, il est possible de simuler efficacement tout aussi bien des systèmes tels que des centres de calculs ou de données que des réseaux pair-à-pair, des grilles ou des clouds. Cette approche a été intégrée à l'outil de simulation open-source SimGrid. Nous montrons que notre solution nous permet de modéliser de tels systèmes bien plus précisément que les autres simulateurs du domaine sans perdre en vitesse de simulation. SimGrid est même dans certaines simulations plusieurs ordres de grandeur plus rapide